Stereo and Shape-from-Shading Cue Fusion for Dense 3D Reconstruction in Endoscopic Surgery

Dense 3D reconstruction of the surgical site is important for providing image-guidance, augmented reality and, in robotic surgery, active constraints. The challenge with endoscopic images is that soft-tissue surfaces do not always have salient characteristics and computational techniques fail to achieve unique correspondence. In this paper, we propose a novel method for handling homogenous regions by fusing visual cues using a combination of Shape-from-Shading (SFS) and stereo. A sparse reconstruction of the underlying structure is performed with a feature-based algorithm and used to initialise and guide two independent SFS modules, which infer monocular dense relative depth. The two reconstructions are then registered with nearest neighbour 3D matching, which is directly translated into a dense 2D disparity estimate. Our only assumption is consistency of the two reconstructions and this is sufficient for the overall scheme to be effective. We validate the approach quantitatively with benchmark phantom data and comparison against the state-ofthe-art endoscopic reconstruction algorithms.